Effect of the Hadamard Transform on Motion Estimation of Different Layers in Video Coding

In video coding, the most commonly used Motion Estimation distortion metrics are predominantly based on the Sum of Absolute Differences (SAD) and the Sum of Absolute Transformed Differences (SATD). Consequently the Joint Model (JM) H.264/AVC Reference Software utilises them and by default, the JM software selects the SAD as the Error Metric for FullPixel (first layer) motion estimation and the SATD as the Error Metric for Half and Quarter-Pixel (second and third layer respectively) motion estimation. Although SATD is much slower than SAD, it more accurately predicts quality from the standpoint of both objective and subjective metrics. In this paper, our experimental results show that the current H.264/AVC Rate-Distortion Optimisation method can have a negative impact when the SATD is applied. More specifically, although the SATD results in a lower bit rate with the same Peak Signal to Noise Ratio (PSNR) when applied in the integer pixel motion estimation with the subpel search disabled, it does not result in a better Rate-Distortion (R-D) performance when applied in the integer pixel motion estimation with the subpel search enabled, when compared to applying the SAD in the integer pixel motion estimation with the subpel search enabled.